Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install app dependencies into a virtual environment #257

Merged
merged 1 commit into from
Aug 30, 2024
Merged

Conversation

edmorley
Copy link
Member

@edmorley edmorley commented Aug 30, 2024

App dependencies are now installed into a Python virtual environment (aka venv / virtualenv) instead of into a custom user site-packages location.

This:

  1. Avoids user site-packages compatibility issues with some packages when using relocated Python (see Switch to using a virtual environment for app dependencies #253)
  2. Improves parity with how dependencies will be installed when using Poetry in the future (since Poetry doesn't support --user installs)
  3. Unblocks being able to move pip into its own layer (see Move pip into its own layer #254)

This approach is possible since pip 22.3+ supports a new --python / PIP_PYTHON option which can be used to make pip operate against a different environment to the one in which it is installed. This allows us to continuing keeping pip in a separate layer to the app dependencies (currently the Python layer, but in a later PR pip will be moved to its own layer).

Now that app dependencies are installed into a venv, we no longer need to make the system site-packages directory read-only to protect against later buildpacks installing into the wrong location.

Note: For a venv to work, it depends upon the <venv_layer>/bin/python symlink being earlier in PATH than the main Python installation. To achieve that with CNBs, the venv's layer name must be alphabetically after the Python layer name. In addition, lifecycle 0.20.1+ is required, since earlier versions didn't implement the spec correctly during the execution of later buildpacks (see buildpacks/lifecycle#1393).

This has been split out of the Poetry PR for easier review.

See also:

Closes #253.
GUS-W-16616226.

App dependencies are now installed into a virtual environment (aka venv
or virtualenv) instead of into a custom user site-packages location.

This:
1. Avoids user site-packages compatibility issues with some packages
   when using relocated Python (see #253)
2. Improves parity with how dependencies will be installed when using
   Poetry in the future (since Poetry doesn't support `--user`)
3. Unblocks being able to move pip into its own layer (see #254)

This approach is possible since pip 22.3+ supports a new `--python`
/ `PIP_PYTHON` option which can be used to make pip operate against
a different environment to the one in which it is installed. This
allow us to continuing keeping pip in a separate layer to the app
dependencies (currently the Python layer, but in a later PR pip will
be moved to its own layer).

Now that app dependencies are installed into a venv, we no longer need
to make the system site-packages directory read-only to protect against
later buildpacks installing into the wrong location.

This has been split out of the Poetry PR for easier review.

See also:
- https://docs.python.org/3/library/venv.html
- https://pip.pypa.io/en/stable/cli/pip/#cmdoption-python

Closes #253.
GUS-W-16616226.
@edmorley edmorley added enhancement New feature or request semver: major labels Aug 30, 2024
@edmorley edmorley self-assigned this Aug 30, 2024
@edmorley edmorley marked this pull request as ready for review August 30, 2024 14:53
@edmorley edmorley requested a review from a team as a code owner August 30, 2024 14:53
@edmorley edmorley enabled auto-merge (squash) August 30, 2024 15:18
edmorley added a commit that referenced this pull request Aug 30, 2024
pip is now installed into its own layer (as a user site-packages
install) instead of into system site-packages in the Python layer.

This is possible now that the user site-packages is no longer being
used for app dependencies, after the switch to venvs in #257.

pip being in its own layer has the following advantages:
1. We can more easily exclude pip from the build/run images when using
   other packages managers (such as for the upcoming Poetry support).
2. A change in pip version no longer unnecessarily invalidates the
   Python layer.
3. In the future we could more easily exclude pip from the run image
   entirely, should we wish (see #255).

This has been split out of the Poetry PR for easier review.

Closes #254.
GUS-W-16616956.
@edmorley edmorley merged commit 43f66bc into main Aug 30, 2024
9 checks passed
@edmorley edmorley deleted the switch-to-venv branch August 30, 2024 16:49
edmorley added a commit that referenced this pull request Aug 30, 2024
pip is now installed into its own layer (as a user site-packages
install) instead of into system site-packages in the Python layer.

This is possible now that the user site-packages is no longer being
used for app dependencies, after the switch to venvs in #257.

pip being in its own layer has the following advantages:
1. We can more easily exclude pip from the build/run images when using
   other packages managers (such as for the upcoming Poetry support).
2. A change in pip version no longer unnecessarily invalidates the
   Python layer.
3. In the future we could more easily exclude pip from the run image
   entirely, should we wish (see #255).

This has been split out of the Poetry PR for easier review.

Closes #254.
GUS-W-16616956.
edmorley added a commit that referenced this pull request Aug 30, 2024
pip is now installed into its own layer (as a user site-packages
install) instead of into system site-packages in the Python layer.

This is possible now that the user site-packages location is no longer
being used for app dependencies, after the switch to venvs in #257.

pip being in its own layer has the following advantages:
1. We can more easily exclude pip from the build/run images when using
   other packages managers (such as for the upcoming Poetry support).
2. A change in pip version no longer unnecessarily invalidates the
   Python layer.
3. In the future we could more easily exclude pip from the run image
   entirely, should we wish (see #255).

This has been split out of the Poetry PR for easier review.

Closes #254.
GUS-W-16616956.
heroku-linguist bot added a commit that referenced this pull request Aug 30, 2024
## heroku/python

### Changed

- App dependencies are now installed into a virtual environment instead of user site-packages. ([#257](#257))
- pip is now installed into its own layer (as a user site-packages install) instead of into system site-packages in the Python layer. ([#258](#258))
edmorley added a commit that referenced this pull request Sep 3, 2024
The Python package manager Poetry is now supported for installing app
dependencies:
https://python-poetry.org

To use Poetry apps must have a `poetry.lock` lockfile, which can be
created by running `poetry lock` locally, after adding Poetry config to
`pyproject.toml` (which can be done either manually or by using
`poetry init`). Apps must only have one package manager file (either
`requirements.txt` or `poetry.lock`, but not both) otherwise the
buildpack will abort the build with an error (which will help prevent
some of the types of support tickets we see in the classic buildpack).

Poetry is installed into a build-only layer, so is not available at
run-time to reduce image size. The app dependencies are installed into
a virtual environment (the same as for pip, after #257), which is on
`PATH` so does not need explicit activation when using the app image.
As such, use of `poetry run` or `poetry shell` is not required at
run-time to use dependencies in the environment.

When using Poetry, pip is not explicitly installed, since Poetry
includes its own bundled copy that it will use instead (for the small
number of Poetry operations for which it still calls out to pip, such
as package uninstalls).

Both the Poetry and app dependencies layers are cached, however, the
Poetry download/wheel cache is not cached, since using it is slower than
caching the dependencies layer (for more details see the comments on
`poetry_dependencies::install_dependencies`).

The `poetry install --sync` command is run using `--only main` so as to
only install the main dependencies group and not any other groups (such
as test/dev/... groups).

Relevant Poetry docs:
- https://python-poetry.org/docs/cli/#install
- https://python-poetry.org/docs/configuration/
- https://python-poetry.org/docs/managing-dependencies/#dependency-groups

Work that will be handled later:
- Support for selecting Python version via `tool.poetry.dependencies.python`:
  #260
- Build output and error messages polish/CX review (this will be
  performed when switching the buildpack to the new logging style).
- More detailed user-facing docs:
  #11

Closes #7.
GUS-W-9607867.
GUS-W-9608286.
GUS-W-9608295.
edmorley added a commit that referenced this pull request Sep 4, 2024
The Python package manager Poetry is now supported for installing app
dependencies:
https://python-poetry.org

To use Poetry, apps must have a `poetry.lock` lockfile, which can be
created by running `poetry lock` locally, after adding Poetry config to
`pyproject.toml` (which can be done either manually or by using
`poetry init`). Apps must only have one package manager file (either
`requirements.txt` or `poetry.lock`, but not both) otherwise the
buildpack will abort the build with an error (which will help prevent
some of the types of support tickets we see in the classic buildpack
with users unknowingly mixing and matching pip + Pipenv).

Poetry is installed into a build-only layer (to reduce the final app
image size), so is not available at run-time. The app dependencies are
installed into a virtual environment (the same as for pip after #257,
for the reasons described in #253), which is on `PATH` so does not need
explicit activation when using the app image. As such, use of
`poetry run` or `poetry shell` is not required at run-time to use
dependencies in the environment.

When using Poetry, pip is not installed (possible thanks to #258), since
Poetry includes its own internal vendored copy that it will use instead
(for the small number of Poetry operations for which it still calls out
to pip, such as package uninstalls).

Both the Poetry and app dependencies layers are cached, however, the
Poetry download/wheel cache is not cached, since using it is slower than
caching the dependencies layer (for more details see the comments on
`poetry_dependencies::install_dependencies`).

The `poetry install --sync` command is run using `--only main` so as to
only install the main `[tool.poetry.dependencies]` dependencies group
from `pyproject.toml`, and not any of the app's other dependency groups
(such as test/dev groups, eg `[tool.poetry.group.test.dependencies]`).

I've marked this `semver: major` since in the (probably unlikely) event
there are any early-adopter projects using this CNB that have both a
`requirements.txt` and `poetry.lock` then this change will cause them to
error (until one of the files is deleted).

Relevant Poetry docs:
- https://python-poetry.org/docs/cli/#install
- https://python-poetry.org/docs/configuration/
- https://python-poetry.org/docs/managing-dependencies/#dependency-groups

Work that will be handled later:
- Support for selecting Python version via `tool.poetry.dependencies.python`:
  #260
- Build output and error messages polish/CX review (this will be performed
  when switching the buildpack to the new logging style).
- More detailed user-facing docs:
  #11

Closes #7.
GUS-W-9607867.
GUS-W-9608286.
GUS-W-9608295.
edmorley added a commit that referenced this pull request Sep 4, 2024
The Python package manager Poetry is now supported for installing app
dependencies:
https://python-poetry.org

To use Poetry, apps must have a `poetry.lock` lockfile, which can be
created by running `poetry lock` locally, after adding Poetry config to
`pyproject.toml` (which can be done either manually or by using
`poetry init`). Apps must only have one package manager file (either
`requirements.txt` or `poetry.lock`, but not both) otherwise the
buildpack will abort the build with an error (which will help prevent
some of the types of support tickets we see in the classic buildpack
with users unknowingly mixing and matching pip + Pipenv).

Poetry is installed into a build-only layer (to reduce the final app
image size), so is not available at run-time. The app dependencies are
installed into a virtual environment (the same as for pip after #257,
for the reasons described in #253), which is on `PATH` so does not need
explicit activation when using the app image. As such, use of
`poetry run` or `poetry shell` is not required at run-time to use
dependencies in the environment.

When using Poetry, pip is not installed (possible thanks to #258), since
Poetry includes its own internal vendored copy that it will use instead
(for the small number of Poetry operations for which it still calls out
to pip, such as package uninstalls).

Both the Poetry and app dependencies layers are cached, however, the
Poetry download/wheel cache is not cached, since using it is slower than
caching the dependencies layer (for more details see the comments on
`poetry_dependencies::install_dependencies`).

The `poetry install --sync` command is run using `--only main` so as to
only install the main `[tool.poetry.dependencies]` dependencies group
from `pyproject.toml`, and not any of the app's other dependency groups
(such as test/dev groups, eg `[tool.poetry.group.test.dependencies]`).

I've marked this `semver: major` since in the (probably unlikely) event
there are any early-adopter projects using this CNB that have both a
`requirements.txt` and `poetry.lock` then this change will cause them to
error (until one of the files is deleted).

Relevant Poetry docs:
- https://python-poetry.org/docs/cli/#install
- https://python-poetry.org/docs/configuration/
- https://python-poetry.org/docs/managing-dependencies/#dependency-groups

Work that will be handled later:
- Support for selecting Python version via `tool.poetry.dependencies.python`:
  #260
- Build output and error messages polish/CX review (this will be performed
  when switching the buildpack to the new logging style).
- More detailed user-facing docs:
  #11

Closes #7.
GUS-W-9607867.
GUS-W-9608286.
GUS-W-9608295.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request semver: major
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch to using a virtual environment for app dependencies
2 participants